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(57) Abstract 

Method and apparatus for computer-aided assessment of 
organizational process or system. Method and apparatus are adapted 
to display questions on a computer to an assessor, who then inputs 
numerical and/or textual inputs relative to the assessor's perception 
of the process or system. Filtering techniques inhibit entry of 
unsupported numerical and/or textual inputs which may be untrue 
and/or exaggerated. Questions used in combination with additional 
questions provide more accurate assessment of the system or process, 
where the question combinations are already existing in a database. 
Evaluations of collected information are performed and additional 
corrective action questions are asked to identify degrees of risk and 
areas of weakness of the process or system. 
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COMPUTER-AIDED METHODS AND APPARATUS FOR ASSESSING AN 
ORGANIZATIONAL PROCESS OR SYSTEM 



CROSS-REFERENCE TO RELATED APPLICATION 
This application is a continuation-in-part of U.S. Patent Application Serial No. 
5 08/351,896, filed on December 8, 1994. 

BACKGROUND OF THE INVENTION 

1 . Field of the Invention 

The invention relates to computer-aided methods and apparatuses for assessing 
organizational processes or systems. 

10 2. Description of the Related Art 

Large entities such as corporations, professional associations, and government units often 
perform organizational assessments both within their own organizations and those of third 
parties, such as corporate divisions, subsidiaries, departments, and third party providers. The 
assessments can cover a wide range of topics, often relating to such matters as safety, efficiency, 

15 cost control, and reliability. Conventionally, such evaluations have been conducted in 
conjunction with on-site audits and inspections. Such audits and inspections, however, tend to 
be burdensome, expensive, and time consuming for both the assessing and audited entities. 

To reduce the burdens associated with these evaluations, surveys are commonly employed 
to gather information concerning organizational processes or systems. A problem with surveys, 

20 however, is that validation of the accuracy and truthfulness of answers received is often difficult 
and expensive, especially when responses are prepared by potentially biased persons, such as 
suppliers of goods and services. 

Another problem with conventional survey techniques is associated with generating an 
optimal structure for the survey. For example, a useful data gathering technique called 

25 "conditional response" involves presenting an assessor a question, and based on the answer to 
the question, branching to one or more subsequent questions. Each subsequent question may 
then also branch to further subsequent questions based on answers provided. In this manner, a 
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complicated "tree" of questions and answers may be prepared. One problem with the conditional 
response technique, however, is that the assessment criteria which direct branching are highly 
subjective. Thus, the person developing the criteria may bias the survey. Furthermore, preparing 
or revising a conditional response survey tends to be difficult and expensive since the "tree" is 

5 highly structured, thus making preparation of the tree, or revisions to the tree, complicated, time 
consuming and cumbersome. For instance, if an early question in the tree is changed, then a 
whole series of subsequent '"branching" questions may also have to be changed. 

A further problem with the conditional response technique is that available answers tend 
to be absolute in nature. For example, responses to the questions typically demand a "yes" or 

10 a "no" with no option for a qualified response. It is often useful, however, to use other types of 
questions demanding nonabsolute responses. For example, a survey may call for responses such 
as numerical responses, multiple choice responses, arbitrary textual responses, or multiple 
choices from a number of selections (e.g. "check all that apply"). Although these nonabsolute 
responses are often useful, adapting them to the conditional response technique often proves 

15 complicated and cumbersome. 

SUMMARY OF THE INVENTION 

A method and apparatus according to various aspects of the present invention uses a 
computer to gather information about an organizational process or system by asking an assessor 
for information about the organizational process or system. Information is gathered by 
20 displaying questions on a computer and storing the assessor's responses. As the information is 
gathered, the responses are analyzed. The analysis performs various functions, such as 
identifying relevant subject matter, identifying appropriate personnel for submitting responses, 
accumulating information and evidence relating to the subject matter, filtering biased or 
exaggerated responses, and identifying potential solutions or improvements. 

25 BRIEF DESCRIPTION OF THE DRAWING 

A more complete understanding of the present invention may be derived by referring to 
the detailed description and claims when considered in connection with the figures, and: 
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Figure 1 depicts an apparatus including a computer, a display device, and an input device. 
Figure 2 depicts a general chart of an assessment process according to various aspects of 

the present invention. 

Figure 3 depicts a general chart of an exemplary assessment initiation system. 
5 Figure 4 depicts a flow-chart wherein an assessor is prompted to input a numerical input 

reflective of how well a system addresses an issue. 

Figure 5 depicts a flow-chart wherein an assessor is prompted to input a numerical input 
reflective of how extensively a system is deployed. 

Figure 6 depicts a flow chart wherein an assessor is prompted to input a numerical input 
1 0 reflective of results achieved by the system. 

Figures 7A-7E depict a series of "sliding bar" displays. 

Figure 8 depicts a flow chart for a corrective action system. 



DETAILED DESCRIPTION OF THE INVENTION 



The subject matter of the present invention is particularly suited for use in connection with 
1 5 methods and apparatus for assessing an organizational process or system, such as computer-aided 
systems for measuring, evaluating, and gathering information about organizational processes and 
systems. As a result, the preferred exemplary embodiment of the present invention is described 
in that context. It should be recognized, however, that the description is not intended as a 
limitation on the use or applicability of the present invention, but is instead provided merely to 
20 enable a full and complete description of a preferred embodiment. 

Referring now to Figure 1 , a computer system suitable for implementing an organizational 
assessment system according to various aspects of the present invention comprises a 
conventional desktop personal computer system 100. The computer 100 may be configured in 
any suitable manner to implement various aspects of the present invention. For example, the 
25 computer 100 may include a memory 1 10 and a processing unit 1 12. The computer 1 00 may be 
further adapted, using any suitable computer software or hardware, to perform the various 
functions as described herein. In the present embodiment, the computer may comprise an IBM 
or Apple compatible computer, such as a Sytech 486166 or a Dell 433SINP. Alternatively, the 
computer 100 comprises any appropriate computer system, such as a mainframe computer, a 
30 minicomputer, a network server, or any other suitable computer system. 



3 
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In the present embodiment, the computer 100 includes various interfaces to operate in 
conjunction with an assessor. For example, the computer 100 is suitably connected via a line 106 
to a display device 102 to provide information to a human assessor, such as a computer screen 
like a Sony Multiscan 17se or a NEC MultiSync 3FGe computer screen. The computer 100 is 
5 also suitably connected via a second line 108 to an input device 104 to facilitate the submission 
of information by the human assessor. The input device 104 of the computer may be a keyboard, 
a mouse, or a touch screen. In some instances the input device 104 may include a speech 
recognition device that converts spoken words into machine-readable inputs. Additional or 
substitute input and output devices, such as modems, printers, speakers, voice recognition 
10 circuitry, or any other suitable input or output device may be connected to the computer 100 to 
facilitate communication with the human assessor or another computer system. 

The computer 100 suitably operates in accordance with at least one program to perform 
various functions in accordance with various aspects of the present invention, such as displaying 
questions, receiving and storing responses, comparing responses to various values, and preparing 
15 reports. A human assessor interacts with the computer and the program, for example in 
conjunction with a series of computer-driven questions. Typically the assessor has at least some 
knowledge about the organizational process or system. At least some of the computer-driven 
questions are adapted to prompt an assessor to submit via the input device 104 a response relating 
to the capability of the organizational process or system to address an issue. 
20 For example, referring now to Figure 2, in accordance with an exemplary embodiment of 

an organizational assessment system according to various aspects of the present invention, a 
suitable organizational assessment system comprises: an assessment initiator system 210; an 
information gathering system 212; and a corrective action system 214. The assessment initiator 
system 210 is suitably initiated first, for example to identify appropriate personnel, subject 
25 matter, general organizational characteristics, or other generalized information. The information 
gathering system 212 may then be implemented, suitably subsequent to and in accordance with 
the results generated by the assessment initiator system 210, to accumulate information relating 
to the relevant processes or systems. The corrective action system 214 may also be initiated, 
suitably subsequent to and in accordance with the results generated by the information gathering 
30 system 212, for example to identify significant problems and potential solutions. Although each 
of these systems suitably operates in conjunction with the others in the present embodiment, each 
of the systems may be configured to operate independently or with less than all of the other 
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systems or features. Further, each of these systems may operate on the same or different 
computers 100 and may interact with the same or different human assessors. 

In the present embodiment, the assessment initiator system 210 is suitably configured to 
identify knowledgeable personnel, acquire general organizational information, and perform any 
5 other function for facilitating the organizational assessment and the proper performance of the 
information gathering system 212 or the corrective action system 214. In the present 
embodiment, the assessment initiator system 210 reads a series of questions from a memory in 
the computer 100 and provides them to a primary human assessor via the display device 102. 
The questions may be formatted as a single series of questions or provided in any other suitable 
10 format, such as in a conditional response format. The primary human assessor is suitably a 
person with broad knowledge relating to the organization to be assessed, such as a company 
president or the head of a division. The assessment initiator 210 may also operate in conjunction 
with multiple human assessors, each providing independent responses and information to the 
questions posed by the assessment initiator system 210. 
1 5 Referring now to Figure 3 . the assessment initiator system 2 1 0 suitably performs multiple 

stages. For example, the assessment initiator system 210 may initially request general 
information about the organization (step 310). Such general information might include the 
organization's name, address, type, size, character of its business, and the like. 

The assessment initiator system 210 may further suitably request information regarding 
20 the characteristics and capabilities of the relevant organization (step 3 12). For example, the 
human assessor may be asked questions such as whether the organization has a manufacturing 
department, a legal department, an accounting department, or the like. Data acquired by the 
assessment initiator system 210 relating to the organizational characteristics may be useful for 
identifying general issues and selecting particular groups of questions to be presented by the 
25 assessment initiator system 210. the information gathering system 212, and the corrective action 
system 214. 

Upon entry of such information by the human assessor, the assessment initiator system 210 
suitably generates at least one question configured to identify personnel with substantive 
knowledge of specific subjects pertinent to the assessment (step 314). Such personnel may 
30 include, for example, high-ranking employees, supervisors, board members, officers or the like. 
For example, a first question may request the name and office location of the director of 
purchasing, and a second question may ask for similar information for the director of human 
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resources. Any number of questions may be posed to identify any number of persons or 
departments likely to possess relevant information. The nature of these questions may be 
adjusted according to previous information submitted by the human assessor. For example, if 
the human assessor previously indicated that the organization has no computer system, questions 
5 for identifying the director of computer systems may be omitted. Similarly, if the human 
assessor indicated that the organization has a legal department, the assessment initiator system 
210 may request further information, such as names and office locations for the general counsel, 
chief patent counsel, and chief litigation counsel. 

An organizational assessment system according to various aspects of the present invention 
10 further suitably implements the information gathering system 212 to gather specific information 
relating to the relevant organization. In the present embodiment, the information gathering 
system 212 operates in conjunction with a database of questions stored in the memory 1 10 of the 
computer 100 to generate at least a portion of a set of questions to be provided to a human 
assessor by the information gathering system 212. Further, information received by the 
1 5 assessment initiator system 210 is suitably used to formulate the set of questions provided to the 
human assessors. For example, if the results of the assessment initiator system 210 indicate that 
the subject organization performs no manufacturing, questions stored in the memory 1 10 relating 
to manufacturing issues may be omitted. 

Generally, the information gathering system 212 poses a series of questions, suitably in a 
20 conditional response format, to acquire information. The questions are suitably posed and the 
responses stored and analyzed, for example to establish the validity and substantiation of the 
responses. For example, a first input may be received from the input device of the computer 
system in response to a first question. The first input is stored in the memory of the computer, 
and the first input may reflect the assessor's perception of the capability of the organizational 
25 process or system to address an issue. 

Once the first input is received, it is suitably compared within a processing unit of the 
computer to a first value. If the first input has a first predetermined characteristic in relation to 
the first value then the method may involve prompting the assessor to identify evidence that 
supports the first input. A "predetermined characteristic" may be denned to mean, for example, 
30 that the input has a value at least as great, greater than, equal to or less than, or less than the first 
value. If the evidence is identified by the assessor that supports the first input, then the method 
mav involve validating the first input for subsequent evaluation. If evidence is not identified that 
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supports the first input, then the method may involve inhibiting validation of the first input until 
the evidence is identified or until the first input is changed to have a second predetermined 
characteristic in relationship to the first value. 

The comparison and validation methods outlined above have the effect of "filtering" data 
5 to remove unsupported, biased, erroneous, or exaggerated data. For instance, an assessor may 
be prevented from entering a favorable rating that is. e.g., above the first value, if the assessor 
cannot identify evidence to support that favorable of a rating. 

In a preferred embodiment, the first input may be a numerical input on selected from a 
scale, for example a 0-100% scale, a 1-3 scale, a 0-10 scale, or a 1-100 scale. This scale may be 
1 0 a sliding bar scale. For instance, the display device may display the following "issue" presented 
in the form of a statement: 

Responsibilities and authorities for all personnel affecting quality are clearly 
defined and documented. 

At this point the assessor may be asked "How well does your process (or system) address 
1 5 this issue?" The assessor may then input on a sliding scale of the computer (e.g. , typically using 
arrow keys on a computer keyboard) his or her perception of how well the organization process 
(or system) addresses the above-referenced issue. 

To illustrate, in one example the assessor might enter in a value of 65%, with 100% being 
the best evaluation and 0% being the worst evaluation. In such example, the first value may be, 
20 for example, 50%. and the first predetermined characteristic may be defined to mean that the first 
numerical input is greater than the first value. In this example, since the first numerical input 
(65%) is greater than the first value (50%), the first numerical input meets the first predetermined 
characteristic, and therefore the assessor may then be prompted to identify evidence that supports 
the first numerical input. Such "evidence" may be in various forms. For instance, it may be 
25 visible evidence that is simply affirmed by the assessor. In other embodiments it may be 
documentary evidence that is identified by name by the assessor. 

In an alternative embodiment of this invention, the first input may be a true/false or yes/no 
input, a numerical input, or a textual input. In addition, the first input may be a selection of only 
one item from a list of several choices. For example, with reference to the above illustration, an 
30 assessor may be asked to select one from the following list in response to the statement: very 
well, adequately, not well, not at all. Alternatively, the first input may be a selection of one or 
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more applicable items from a list of several choices. 

In one embodiment the "evidence' may be affirmed by the assessor if the assessor indicates 
that the process or system is demonstrable (i.e., can be demonstrated). If the process or system 
is demonstrable, then the first numerical input may be validated for subsequent evaluation. If 
5 the system or process is not demonstrable, then validation of the first numerical input may be 
inhibited until the assessor indicates that the system or process is demonstrable or until the 
assessor changes the first numerical input to have a second predetermined characteristic in 
relationship to the first value. In this context, "validation" means that the numerical input is 
accepted by or into the computer for future evaluation. 
10 In some embodiments, if evidence is not identified that supports the first input, then the 

display of subsequent questions may be inhibited until the evidence is identified or until the first 
input is changed to have the second predetermined characteristic in relation to the first value. 

In an embodiment of the invention the "second predetermined characteristic" may be 
defined to mean that the first input is less than the first value. In the example given above, if the 
1 5 first value is 50% and the assessor indicates that the first input is 65%, then the assessor may be 
asked to indicate that the process or system is demonstrable. If the assessor does so, then 
validation and/or display of subsequent questions proceeds. If the assessor does not, then 
validation may be inhibited, and/or the display of subsequent questions may be inhibited, until 
the assessor indicates that the process or system is demonstrable, or until the assessor changes 
20 the first input to be below 50% (e.g., 45%). 

In an embodiment of the invention further computer-driven questions may be displayed 
on the display device. These computer-driven questions may be adapted to prompt the assessor 
to input a second input on the input device of the computer. The second input may reflect the 
assessors perception of how extensively the organizational process or system is deployed (i.e., 
25 how extensively the organizational process or system is actually used). For instance, after the 
first input is validated, the assessor may be asked to indicate how well the process being assessed 
is deployed. At this point the assessor may again answer a value (e.g., a percentage value on a 
0-100 scale) that is indicative of how well the system or process is deployed. The term "how 
extensively the organizational process or system is deployed" refers to an assessment of how 
30 extensively processes or systems are actually deployed versus theoretically deployed. 

The capability of an organizational process or system to address an issue refers to an 
assessment as to the ability of the theoretical organizational process or system to address an 
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issue. In other words, the first input reflects an assessment as to whether the organizational 
process or system can address an issue. The second input then reflects how extensively the 
organizational process or system is actually deployed to address that issue. In other words, does 
the assessed process or system actually operate or does it sit on the shelf and/or only theoretically 
5 operate? For instance, a process or system may have a high (e.g., 90%) first input indicating that 
the process or system has a favorable ability to address the issue, but the second input may 
relatively low (e.g., 30%), indicating that process or system is not widely deployed. 

Superior results have been achieved by prompting these questions together in conjunction 
with filtering techniques (see. e.g. Table 2 and accompanying text below). The results are 
10 achievable because an apparatus adapted to prompt these combinations of questions in 
conjunction with filtering techniques can address the following two phenomena frequently 
encountered when assessing a process or system: (1) a process or system is not supported by 
evidence but the users nevertheless use this process or system, or (2) the process or system is 
supported by evidence but it is not used. If either of these phenomena are present, then the first 
15 input relating to the assessed process or system may be dependent on knowledge embedded 
within workers. These workers, however, may quit, retire, be promoted, or otherwise cease 
performing, all without "passing on" the knowledge which enabled the assessed process or 
system to achieve the first input. Thus the stability of the first input is questionable if either of 
these two phenomena are present. 
20 The filtering techniques provide information concerning the first phenomenon. As 

described above, these filtering techniques also tend to inhibit favorable first inputs that are 
unsupported. Thus these filtering techniques also increase the accuracy of the first input if the 
first phenomenon is present. The second input provides information concerning the second 
phenomenon. This information can be used with the first input in subsequent evaluations (e.g., 
25 by mathematical manipulation such as multiplying the first input by the second input to obtain 
a combined input value). Thus an apparatus adapted to prompt the first and second numerical 
inputs, in combination with the filtering techniques, has the effect of providing information 
about, and/or increasing the accuracy of, the first input. The above-described information that 
is provided "about the first input" can be reflective of the stability of the first input. 
30 In the above-described manner an assessor may be prompted to provide filtered numerical 

inputs that can together be used to evaluate the process or system. For instance, a first numerical 
input may be multiplied by a second numerical input (e.g., 90% X 30% = 27%) to provide a 
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multiplied evaluation factor that combines the characteristics of both inputs. Alternately, a 
plurality of first numerical inputs from various assessors may be used, and one or more standard 
deviations for the first and second numerical inputs may be calculated and compared or used in 
combination with each other for evaluation purposes. A wide variety of issues may be surveyed 
5 by a wide variety of assessors. See Table 1 for issues that may be presented. 

Many types of filtering techniques may be used in evaluating the inputs. For example, any 
sort of mathematical manipulations can be used, include multiplying, adding, subtracting, 
dividing, calculating standard deviations between, the first numerical input with the second 
numerical input, or vice versa. 
10 The results of the filtering techniques may be evaluated to determine problem areas that 

might warrant on-site investigation. For instance, if 100 different issues were assessed by 10 
different assessors, the average multiplied evaluation factor might be 55 %, with only 8 issues 
being below 45 %. On-site evaluation teams might then be directed to focus their efforts on 
investigating the 8 issues that were below 45% instead of all 100 issues, thereby saving 
1 5 manpower and expense. Alternately, the on-site teams could be directed to investigate issues that 
had standard deviations above a certain value {e.g., 8%). Again, the efforts of the on-site team 
would be focused and reduced, thereby saving manpower and expense. 

Information or data collected may be further driven or sent to a printer or data collection 
device such as a memory in a computer. Further evaluation may be achieved by having one 
20 computer or control system automatically interact with another, wherein the second computer 
or control system is requested to gather or retrieve further data for validation or evaluation 
purposes. For instance, the second computer or control system may be adapted to provide 
information that verifies or confirms that numerical inputs are reasonably accurate. 

Figure 4 demonstrates an embodiment of the invention which includes an assessment 
25 screen adapted to display a question in the form or a statement or issue. For the purposes of this 
patent it is to be understood that a "question" may be presented in multiple ways. For instance, 
a question may be a standard "yes/no" or "true/false" answer form {e.g., Does the ABC system 
maintain on-time performance records?). In addition, the question may be presented by making 
a statement, or stating an issue, and then asking the assessor to evaluate the statement or issue 
30 on a numerical scale. For instance, the assessor may be presented with the statement that "the 
ABC system maintains on-time performance records." In response thereto, the assessor may 
enter a value on an analog scale reflecting the accuracy of the statement (e.g., 6 on a 1-10 scale, 
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with 10 being the most favorable rating). 

In Figure 4 a series of statements or issues (such as those depicted in Table 1) are shown 
on the assessment screen. The following question is then displayed: 



"How well does your system address this issue?" 

5 At this point, the assessor is prompted to input a numerical input, on an analog percentage scale, 
which reflects how well the assessed system addresses the issue. As shown by the arrows in 
Figure 4, if the rating is less than 50%. then the assessor may be asked if the system is 
demonstrable. If the answer is "no," then additional displays may be shown as represented by 
Box X in Figure 4. If more questions are scheduled to be presented to the assessor, then the 
10 process or apparatus may proceed to the next question for assessment. If no more questions are 
scheduled, then the data may be saved and the assessor may be returned to a user category menu. 

To illustrate, in Figure 4. the apparatus may be adapted such that the first value is 50% and 
the first characteristic in relationship to the first value is whether the first numerical input (i.e., 
the initial rating) is greater than 50%. The second characteristic in relationship to the first value 
15 is whether the first numerical input is less than 50%. The "evidence" is whether the assessed 
system is demonstrable (i.e., can the system be demonstrated?). Subsequent evidence may be 
requested in the form of supporting documentation. For instance, if the assessor indicates that 
the assessed system is demonstrable, then the assessor may be asked to indicate whether 
supporting documentation exists. If supporting documentation exists, then the assessor may be 
20 asked to indicate document names. If supporting documentation does not exist, then the assessor 

may proceed to Box X. 

In Figure 4, the first numerical input may be compared to other values besides the first 
value (the first value in this application acts as a trigger point, or set point). For instance, the first 
numerical input may be compared to determine if it is between 50-70%. If the first numerical 
25 input is between 50% and 70%, then the assessor may be asked to indicate whether the system 
is demonstrable, as shown in Figure 4. If the system is not demonstrable, then the assessor may 
be asked to change the rating. In other words, the system shown in Figure 4 may inhibit display 
of subsequent questions if evidence (e.g., system demonstrably) is not presented that will 
support ratings in the range of 50-70%. In Figure 4, if the first numerical input is between 
50-70% and the system is demonstrable, then the assessor is asked further questions as outlined 



30 
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above for Figure 4 (e.g., if supporting documentation exists, etc.). 

The system in Figure 4 also has the capability of comparing the first numerical input to a 
third value. As shown in L if the first numerical input is 70% or greater, then the assessor is 
asked to determine if the system is demonstrable. If the system is not demonstrable, then the 
apparatus inhibits display of subsequent questions until the assessor changes the rating (i.e., 
lowers the first numerical input below 70%). Furthermore, the assessor is subsequently asked 
if supporting documentation exists. If the assessor indicates that no supporting documentation 
exists, then again the display of subsequent questions is inhibited until the assessor indicates that 
supporting documentation exists or changes the first numerical input to a lesser value. 

In Figure 4, Box X may indicate that the first numerical input is validated. In this context 
"validation" may simply mean that the first numerical input is stored, that the assessor is not 
inhibited from proceeding to subsequent questions, and/or that the display of subsequent 
questions is not inhibited. 

The apparatus shown in Figure 4 has the effect of "filtering" assessment indicia. As shown 
in Figure 4, as the rating (/.<?., numerical input) becomes more favorable (i.e., higher), then 
greater levels of evidentiary support are required. If the rating is less than 50%, then the assessor 
may still proceed to Box X even if the assessed system is not merely demonstrable. Of course, 
if the assessed system is demonstrable, then information is received by the apparatus, and if the 
supporting documentation exists, that information is also received by the apparatus. If the first 
numerical input is between 50-70%, then a greater level of evidence is required before the 
assessor can proceed to Box X. Specifically, at least system demonstrability must be indicated. 
Otherwise the assessor must change (i.e., lower) the first numerical input. Again, if supporting 
documentation exists, the information is received by the apparatus. Finally, if the numerical 
input is 70% or greater, then system demonstrability and supporting documentation must exist. 
Otherwise the assessor must change the first numerical input to a lesser value. 

In effect, the apparatus shown in Figure 4 filters the data collected by inhibiting 
validation/collection of exaggerated, untrue, and/or unsupported numerical inputs. Furthermore, 
as a numerical input becomes more favorable, then the level of support required for that rating 
also increases. 

The numerical input shown in Figure 4 may be modified based on a wide variety of factors 
preferred by the apparatus designers. For instance, in some apparatus lower ratings may be 
preferred (e.g., a rating system of 1-10, where 1 is the most favorable rating). In such case. 
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appropriate revision of the levels of evidence as correlated to numerical inputs would be required. 
In some instances numerical inputs may be utilized to indicate specific events or circumstances 
(e.g., l=daily, 2=weekly, 3=monthly, 4=yearly, or l=always, 2=sometimes, 3=never, etc.) 

In Figure 4, Box X may represent a series of subsequent questions presented to the 
5 assessor. For instance, Box X may be represented by the system shown in Figure 7. In Figure 
7 a question may be displayed on a display device which would ask for a second numerical input 
reflecting the assessor's perception of how extensively the organizational process or system is 
deployed. Again, a filtering apparatus may be employed in conjunction with the second 
numerical input to correlate more favorable ratings with varying levels of evidence. This 
10 filtering apparatus may be similar to the Figure 4 filtering apparatus. Depending on the level of 
the second numerical input, then varying levels of evidence may be required to arrive at Box Y. 
Box Y may serve similar purposes as Box X. That is, it may serve to validate the second 
numerical input, or represent subsequent questions to be displayed to the assessor. 

Figure 6 depicts a system analogous to that of Figure 4, except the initial questions is: 

15 "How well would you rate the results achieved by the system?" 

In Figure 6. a third numerical input is input into the system, the third numerical input 
reflecting the assessor's perception of the results achieved by the organizational process or 
system. A filtering apparatus similar to the filtering system shown in Figure 4 may also be 
employed. Box Z in Figure 6 may represent subsequent questions to be asked. 

20 Preferred embodiments of the invention may employ the apparatus of Figure 4 with the 

apparatus of Figure 7, the apparatus of Figure 4 with the apparatus of Figure 6, the apparatus of 
Figure 4 with the apparatus of Figures 2 and 3, and the apparatus of Figure 7 with the apparatus 
of Figure 6. Furthermore, the order of the apparatus may be reversed or mixed. For example, 
questions presented in the apparatus of Figure 7 may be asked first followed by questions in the 

25 apparatus of Figure 4. 

An apparatus of the invention combining Figure 4 with Figure 7 may provide superior 
evaluation results. It is believed that the superior results may be achieved because the apparatus 
of Figure 1 provides an assessment as to how well a system may address an issue, however it 
doesn't provide an assessment on how extensively the system is deployed. Thus a system may 

30 work (Le.n address an issue) particularly well (Le. 9 greater than 90%); however it may only 

13 

BNSDOC1D: <WO 9952039A 1_l_> 



WO 99/52039 PCT/US99/07505 

minimally deployed. By using Figure 4 and Figure 7. an assessment as to the capability of the 
system combined with an assessment as to system deployment may be achieved. For instance, 
if the system addresses the issue particularly well (i.e., at 90%) but is only 60% deployed, then 
in one embodiment a combined assessment of 90% x 60% = 54% may be calculated. Such 
combined assessments may be particularly helpful to focus on-site assessment team efforts. 

Figures 7A-7E depict a series of screen displays for the VIRTUAL ASSESSOR (formerly 
MAXUS and MAXSYS) process/apparatus provided by Tech-Metrics International, Inc 
(Scottsdale, Arizona). As shown in Figure 7A, the following initial "issue" or statement within 
the category "management responsibility" is displayed: "Quality policy is communicated, 
understood and maintained throughout the organization." The assessor is then prompted to input, 
on a sliding bar 0-1 00% scale, a first numerical input that reflects his or her perception as to how 
well the assessed process (or system) addresses this issue. As shown in Figure 7B, the assessor 
in this example input a first numerical input of 58%. 

At this point the assessor is prompted (/.<?., the word "yes" is highlighted) to indicate 
whether the system is demonstrable or not. Such indication may be made by the assessor 
pressing the "enter ' key when "yes" is highlighted. To indicate "no" the assessor may use an 
input device to move the highlight so that "no" is highlighted instead of "yes" and then pressing 
the "enter" key. If the first numerical input of 58% is higher than a first value (e.g. , higher than 
50%), then the assessor may be required to indicate that the assessed system is demonstrable. 
Otherwise the apparatus may inhibit display of subsequent questions and/or prompt the assessor 
to provide a new (and in this case a lower and less favorable) first numerical input. In this 
example the assessor indicated "yes" so subsequent questions were not inhibited and the assessor 
was not asked to input a new numerical input. 

As shown in Figure 7C, the apparatus then prompted the assessor to indicate whether 
supporting documentation existed. If the first numerical input (here, 58%) is higher than a 
second value (e.g., higher than 60%), then the assessor may be required to indicate that 
supporting documentation exists. Otherwise the apparatus may inhibit display of subsequent 
questions and/or prompt the assessor to provide a new (and in this case a lower and less 
favorable) first numerical input. In this example the assessor indicated "yes" so, as shown in 
Figure 7D, the apparatus prompted the assessor to identify documents by providing document 
names. 

The "system is demonstrable" and "supporting documentation exists" statements, and their 
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associated "yes or no" questions are filtering mechanisms to help inhibit/prevent collection of 
exaggerated or untrue data. Using known empirically gathered information, the apparatus may 
be adapted to have first, second, or subsequent values that serve as "trigger" or "set" points to 
require higher levels of evidentiary support. The theoretical basis for this apparatus is that, on 
5 average, filtered perceptions of quality, as indicated by relatively high first numerical inputs, 
reflect actual conditions. For instance, in the example shown above, a system must be 
demonstrable to support a first numerical input rating greater than the first value (here, greater 
than 50%), and then the system must have supporting documentation to support a first numerical 
input rating higher than a second value (e.g., higher than 60%). 
10 After entering at least one document name in Figure 7D. the display screen shown in 

Figure 7E may be displayed. Figure 7E prompts the assessor to input a second numerical input 
which reflects the assessor's perception of how well the process is deployed. In other words, the 
assessor inputs a numerical evaluation of how extensively the assessed process (or system) is 
actually used or deployed. 
15 Upon completion of the information gathering system 212, the corrective action system 

214 is suitably initiated. The corrective action system 214 suitably analyzes the data 
accumulated by the information gathering system 212 to identify problem areas and explore 
resolution of the problems. Referring now to Figure 8, the corrective action system suitably 
initially analyzes the data gathered by the information gathering system 212 to detect problem 
20 areas (step 810). For example, the data for each of the fields, i.e., the topics addressed by the 
information gathering system 212, may be summarized into a single value for the field. The 
summary may be generated in any manner, such as statistical methods like finding the mean, 
median, or mode of numerical responses. To identify specific problem fields, the summarized 
results are compared to selected thresholds (step 812). The thresholds may be selected in any 
25 suitable manner, for example approximating industry average values for such fields or minimum 
values required to satisfy particular evaluation criteria. 

If all of the summary values for the fields exceed the selected thresholds, the corrective 
action 214 system may terminate. Alternatively, the thresholds may be raised to a level 
corresponding to an improved state of operation (step 814) and the results analyzed again in view 
30 of the new set of thresholds (step 816). In any event, if any of the summarized values fail to 
exceed the selected thresholds, the particular fields are identified for further analysis (step 818). 
Following identification of the relevant fields, the corrective action system 214 generates 
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a new set of questions for each of the relevant fields and poses them to the appropriate personnel. 
In the present embodiment, the appropriate personnel are likely to be the personnel identified in 
conjunction with the application initiator system 210 as being in charge of the departments 
associated with the relevant fields. The questions are suitably designed to assess the magnitude 
5 of the problem, the cause of the problem, and solutions to the problem. For example, a first set 
of questions require designation of a responsible person or persons for addressing the problem 
(step 820). A next set of questions may be designed to determine optimal short-term solutions 
to the problem to be implemented by the designated responsible person (step 822). 

After identifying short-term factors, the corrective action system 214 suitably provides 

10 questions relating to a more complete solution of the problem. For example, a set of further 
questions may be presented to determine the magnitude of the risk and/or potential impact 
associated with the problems in the field (step 824). Additional questions are suitably presented 
to identify the fundamental causes of the problem (step 826) and potential solutions for 
neutralizing those causes and remedying the problem (step 828). Finally, the corrective action 

15 system 214 suitably implements a calendaring set of questions configured to interactively 
generate a timetable for resolution of the problem and performance of a supplementary 
organizational assessment within the problem fields (step 830). 

Once the assessment information is collected and evaluated, reports are created 
summarizing the information, suitably by the computer 100. For example, these reports might 

20 identify an organization's strengths and weaknesses. The report might also suggest guidelines 
for the organization to improve the weaknesses, for example in accordance with the information 
accumulated by the corrective action module 214. In the present embodiment, the report 
generating process may include information relating to the correlation of the respondents* 
responses, a summary of the respondents' views relating to various issues, additional sources of 

25 data identified by the respondents, or any other information which may be suitably drawn from 
or generated in conjunction with the information obtained by the organizational assessment 
system. 

Although the present invention is directed towards computer-driven assessments, 
computers may not be available to the assessors who need to answer questions. Therefore, an 
30 organizational assessment system according to various aspects of the present invention may be 
configured to generate a set of questions suitable for printing to hard copy for use. Preferably, 
the particular selection of questions is generated following a computer-driven implementation 
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of the assessment initiator system 210. Based on the responses provided by the human assessor 
to the assessment initiator system 210, a selection of questions may be derived from a collection 
of questions stored in the memory 1 10 of the computer 100. Further human assessors are given 
hard copies of the questions and answer sheets. The answers sheets typically contain numbers 
5 which correspond to the question numbers. Next to the answer sheet numbers are a plurality of 
circles or other indicia which correspond to the answer choices presented with each question. 
To answer a question, the assessor darkens the circle or otherwise marks or designates the area 
which corresponds to the appropriate answer. The answers sheet can then be scanned into the 
computer and the assessment, including the information gathering system 212 and the corrective 
1 0 action system 214, can be continued on the computer. 

Experiment 

Table 2 reflects experimental information collected to verify the ability of an embodiment 
of the invention to accurately access an organizational system. In this experiment a corporation 
desired to access five external manufacturing locations operated by five of its suppliers of 

15 goods/services. The process/apparatus of the invention was used, as described above in 
conjunction with Table 1, Figure 1, and Figures 7A-7E, to assess these locations. The results for 
such assessment is shown in the columns labeled TMI in Table 2. The corporation then 
conducted manual on-site assessments of the five suppliers. The results of the corporation's 
assessments for these five suppliers is shown in Table 2 under the columns labeled EML1, 

20 EML2, EML3, EML4, and EML5. The difference between the corporation's assessments and 
the assessments prepared using an embodiment of the invention are shown in Table 2 under the 
columns labeled "+/-." As shown in the last row of Table 2, the average difference between these 
assessments was -2.27%, thus indicating that, on average, the assessments using an embodiment 
of the invention were only 2.27% different from the assessments obtained using manual on-site 

25 methods. Since the manual on-site methods are much more expensive and time consuming, these 
results in Table 2 indicate that an embodiment of the invention may obtain comparable results 
with a greatly reduced cost. Moreover, the assessments prepared using an embodiment of the 
invention were slightly lower than those prepared using on-site manual inspections, thus 
producing a slightly more conservative assessment. 

30 In an embodiment of the invention the apparatus and process of the invention may be 

adapted to compare numerical inputs (relating to a plurality of issues, or categories of issues) for 
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a plurality of suppliers. In this manner an objective, relatively quick, and inexpensive assessment 
of the suppliers may be completed. In this manner industry standards may be established, and 
on-site evaluations may be directed to focus their inspection efforts in areas that are below, or 
not within, industry norms. 
5 Although not specifically illustrated in the Figures, it is understood that other additional 

and necessary materials, apparatus, and structural components will be provided, and that these 
and all of the components described above may be arranged and supported in an appropriate 
fashion to form a complete and operative apparatus or process. It is also understood that 
variations may be made in the present invention without departing from the spirit and scope of 
10 the invention. 
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TABLE 1 




4.1 MANAGEMENT RESPONSIBILITY 




1. 


Quality policy is communicated, understood and maintained throughout 
the organization. (I.4. 1.1) 


5 


2. 


Responsibilities and authorities for all personnel affecting quality are 
clearly defined and documented. (1.4.1.2) 




3.1 


Authority delegated to personnel to prevent nonconformity reoccurrence. 
(1.4.1.2) 


10 


3.2 


Authority delegated to personnel to identify & record quality problems. 
(1.4.1.2) 




3.3 


Authority delegated to personnel to initiate & verify corrective action. 
(1.4.1.2) 




3.4 


Authority delegated to personnel to control further processing. (1.4.1 .2) 


15 


4. 


A multi-disciplinary approach is used in the design process with direct 
input in decision making. (1.4.1.2) 




5. 


Periodic top management review of quality system effectiveness is 
supported by appropriate records. (1.4.1.3) 




6. 


Qualified technical personnel are available for design, process, product 
and service support. (1.4.1.2.2) 


20 


7. 


Management representative with authority & responsibility to ensure 
standards compliance (i.e. ISO-9000, QS-9000, NQA, etc.) is clearly 
defined. (1.4.1.2.3) 


25 


8.1 


Documented business plan(s) consider standard's requirements (i.e. 
ISO-9000, QS-9000, NQA, etc.) including competitive product analysis, 
(as applicable) (1.4.1 .4) 




8.2 


Documented business plan(s) considers the standard's requirements 
(i.e. ISO-9000, QS-9000, NQA, etc.) and benchmarking, (as applicable) 
(1.4.1.4) 


30 


8.3 


Documented business pian(s) considers standard's requirements (i.e. 
ISO-9000, QS-9000, NQA, etc.) and R&D plans, (as applicable) (1.4.1.4) 
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TABLE 1 




8.4 


Documented business plan(s) considers standard's requirements (i.e. 
lSO-9000, QS-9000, NQA, etc.) of internal quality and operational 
performance measures (as applicable) (1.4.1.4) 


5 


9. 


Data is used to focus on competitors and/or appropriate benchmarks for 
improving quality, productivity, and operation efficiency. (1.4.1.5) 




10. 


Documented and objective processes are used to measure customer 
satisfaction, (e.g.: a plan with short and long term actions addressing 
customer dissatisfaction factors.) (1.4.1.6) 


i c\ 


11. 


Cross-functional teams are used for the quality planning process. 
(1.4.2.1) 




4.2 QUALITY SYSTEM 




1. 


Quality Manual adequately meets QS-9000 requirements for 
documentation of a comprehensive quality system. 


15 


2.1 


Quality planning process is consistent with the elements of the quality 
system that addresses Product Program plan preparation. (I.4.2.3) 




2.2 


Quality planning process is consistent with the elements of the quality 
system that addresses identification and acquisition of the appropriate 
resources. (I.4.2.3) 


20 


2.3 


Quality planning process is consistent with the elements of the quality 
system and addresses conducting design and process compatibility 
studies. (I.4.2.3) 




2.4 


Quality planning process is consistent with the elements of the quality 
system and addresses updating and maintenance of all quality control 
and inspection methodology. (I.4.2.3) 


25 


2.5 


Quality planning process is consistent with the elements of the quality 
system and addresses identification of suitable verification at appropriate 
stages. (I.4.2.3) 




2.6 


Quality planning process is consistent with the elements of the quality 
system and addresses preparation of control plans and FMEAs. (I.4.2.3) 


30 


2.7 


Quality planning process is consistent with the elements of the quality 
system and addresses review of standards and specifications. (I.4.2.3) 
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TABLE 1 




3. 


Feasibility reviews are conducted to confirm the compatibility of design 
with the manufacturing process, including capacity planning and 
utilization. (I.4.2.3) 


5 


4. 


Engineering requirements are met at the required statistical process 
capability. (I.4.2.3) 




5. 


Control plans are developed to the subsystem, component, and/or 
material level. (I.4.2.3) 




6. 


Control plans include all special characteristics, related process and 
parameters and are identified as such. (1.4.2.3) 


10 


7. 


Control plans are revised when appropriate for product and process 
changes or when processes are found to be unstable or non-capable. 
(1.4.2.3) 




8. 


Control plans cover three phases: prototype, pre-launch, production 
(unless exempted by the customer). (1.4.2.3) 


15 


9. 


Process FMEAs consider all special characteristics. (1.4.2.3) 




10. 


Adequate supporting procedures exist for each element of the quality 
manual. (4.2.2) 




11. 


Special characteristics have been identified and included in the Control 
Plan(s). (I.4.2.3.a) 


20 


12. 


A comprehensive quality system (appropriate to the product or service 
produced) is established and implemented. (4.2.1) 




4.3 CONTRACT REVIEW 


25 


1. 


Contract review activities are adequately documented and maintained to 
ensure that order requirements are understood and are within the 
supplier's capability prior to order acceptance. (I.4.3.2) 




2. 


Standard's requirements (i.e. ISO-9000, QS-9000, NQA, etc.) and 
customer contract requirements are deployed into the quality system. 
(I.4.3.2) 


30 


3. 


Provisions to document and deploy contract changes throughout the 
organization exist (I.4.3.3) 




4. 


Contract review records are maintained. (I.4.3.4) 
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TABLE 1 


4.4 DESIGN CONTROL 


1. 


Design plans for each project have been established and responsibility 
assigned. (I.4.4.2) 


2. 


Responsible personnel are experienced in the "required skills" or 
appropriate equivalents. (L4.4.2) 


3. 


Applicable statutory and regulatory requirements are identified. (I.4.4.4) 


4.1 


Appropriate resources and facilities are available to use computer aided 
design, engineering and analysis. (I.4.4.4) 


4.2 


Technical leadership is provided when CAD/CAE is sub-contracted. 
(I.4.4.4) 


5. 


Formal documented design reviews are conducted per the design plan. 
(I.4.4.5) 


6. 


Design output is documented and expressed as requirements that can 
be verified. (I.4.4.6) 


7.1 


Design output meets design input requirements. (I.4.4.6) 


7.2 


Design output contains or references acceptance criteria. (I.4.4.6) 


7.3 


Design output includes a review of design output documents before 
release. (1. 4.4. 6) 


8.1 


Design outputs are the result of a process that used the "Design 
Techniques" or alternatives? (I.4.4.2) & (I.4.4.6) 


8.2 


Design outputs are the result of a process that used Geometric 
Dimensioning and Tolerancing (GDT). (I.4.4.6) 


8.3 


Design outputs are the result of a process that used analysis of cost / 
performance / risk trade-offs. (I.4.4.6) 


8.4 


Design outputs are the result of a process that used feedback from 
testing, production and the field. (1. 4.4.6) 


8.5 


Design outputs are the result of a process that used analysis of design 
failure mode and effects (DFMEA). (I.4.4.6) 


9. 


Performance testing (life, durability, reliability) is tracked for timely 
completion and conformance. (I.4.4.7) 



22 



BNSDOCID: <WO 9952039A1 J_> 



WO 99/52039 



PCT7US99/07505 





TABLE 1 




10. 


A comprehensive prototype program exists (unless waived by the 
customer or made unnecessary by the generic nature of the product 
supplied). (I.4.4.7) 




11. 


Design validation has been performed at the specified frequencies, 


5 




results recorded, and failures addressed. (I.4.4.8) 




12. 


Design changes are documented and approved by authorized personnel 
before implementation. (I.4.4.9) 




13. 


Written customer approval or waiver has teen obtained prior to a design 
change being implemented into production. (I.4.4.9) 


10 


4.5 DOCUMENT AND DATA CONTROL 




1. 


New and revised documents are reviewed and approved by authorized 
personnel prior to issue. (I.4.5.2) 




2. 


A master list (or equivalent) identifies document revision status. (I.4.5.2) 




3. 


Timely review, distribution and implementation of customer engineering 


15 




standards, specifications and changes. (I.4.5.2) 




4. 


All referenced documents are available on-site. (1.4.5.1) 




5. 


Special characteristic symbols or notations are shown on process control 
plans and similar documents. (1.4.5.1) 




6. 


Where documents or data is retained on software, appropriate controls 


20 




are maintained for changes. (1.4.5.1 & 3) 




4.6 PURCHASING 




1. 


Subcontractors are evaluated and selected based on their ability to meet 
quality system and quality assurance requirements. (I.4.6.2.a) 




2. 


Appropriate level of control over subcontractors is maintained. (I.4.6.2.b) 


25 


3. 


Quality records of subcontractors are kept up to date and used to 
evaluate performance. (I.4.6.2.c) 




4. 


Subcontractor development is conducted using the standard's 
requirements (i.e. ISO-9000, QSR (Sections I & II), NQA, etc.) as the 
fundamental quality system requirement. (1. 4.6.2) 


30 


5. 


Purchasing documents contain data that clearly describe the product or 
service being ordered. (1. 4. 6. 3) 
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6. 


Where applicable, there is provision for the customer (or representative) 
to verify subcontractor quality on the subcontractor's premises. 
(I.4.6.4.2) 


4.7 CONTROL OF CUSTOMER SUPPLIED PRODUCT 


1. 


Material is examined upon receipt to check quantity, identity, and transit 
damage. (L4.7) 


2. 


Material is periodically inspected to detect signs of deterioration, proper 
conditions & storage time limitations. (I.4.7) 


3. 


For product that is lost, damaged or otherwise unsuitable for use, 
records are maintained and reports provided to the customer. (I.4.7) 


4.8 PRODUCT IDENTIFICATION AND TRACEABILITY 


1. 


Product is identified, where appropriate, at alt production stages? (I.4.8) 


2. 


Traceability is maintained and recorded when required by the customer? 
(i.4.8) 


4.9 PROCESS CONTROL 


1.1 


Documented job instructions have been developed and are accessible at 
each work station. (I.4.9) 


1.2 


Documented job instructions communicate requirements to all 
employees involved. (I.4.9) 


1.3 


Documented job instructions provide for verification of job set-ups and 
tool change intervals. (I.4.9) 


1.4 


Documented job instructions specify monitoring of special 
characteristics. (1. 4. 9) 


1.5 


Documented job instructions list requirements for inspection, testing, 
gaging and recording results. (I.4.9) 


1.6 


Documented job instructions provide sample size and frequency. (I.4.9) 


1.7 


Documented job instructions establish approval and rejection criteria. 
(I.4.9) 


1.8 


Documented job instructions list required tools and gages (with 
mastering at required frequency). (I.4.9) 
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1.9 


Documented job instructions describe the identification and handling of 
non-conforming material. (I.4.9) 


5 


1.10 


Documented job instructions specify appropriate notifications and 
corrective actions (including plans for unstable/non-capable processes). 
0-4.9) 




1.11 


Documented job instructions specify application of statistical methods 
required by control plans. (I-4.9) 


10 


1.12 


Documented job instructions identify relevant engineering and 
manufacturing standards and the latest engineering change affecting the 
instruction. (I.4.9) 




1.13 


Documented job instructions display appropriate approvals and dates. 
(14.9) 




1.14 


Documented job instructions display operation name and number. (1.4.9) 




1.15 


Documented job instructions are keyed to process flow charts. (1.4.9) 


15 


1.16 


Documented job instructions show part name and number. (1.4.9) 




1.17 


Documented job instructions show revision date for instructions. (1.4.9) 




1.18 


Documented job instructions define visual controls. (1.4.9) 




2. 


Employees perform operations/inspections according to documented 
instructions. 


20 


3.1 


Process control requirements are met. 




3.2 


The customer's preliminary process capability requirements are met. 
(1.4.9.2) 




3.3 


The customer's ongoing process performance requirements are met. 
(1.4.9.3) 


25 


3.4 


Special causes of variation are investigated and appropriate actions 
taken. (1.4.9.3) 




3.5 


Control charts are annotated with significant process events. (1.4.9.3) 




3.6 


Control charts are maintained and reviewed with highest priority given to 
special characteristics. (1.4.9.3) 
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4.1 


Planned preventive maintenance system includes a maintenance 

schedule established with specific responsibilities assigned. (I.4.9.g) 




4.2 


Planned preventive maintenance system is evaluated for process 
capability improvement. (I.4.9.g) 




4.3 


Planned preventive maintenance system is evaluated for reduction of 
machine/process downtime. (1.4. 9.g) 




4.4 


Maintenance is conducted at the prescribed frequencies for all 
equipment. (1.4. 9. g) 


10 


4.5 


Planned preventive maintenance system tracks availability of 
replacement parts for key manufacturing equipment. (I.4.9.g) 




4.6 


Planned preventive maintenance system uses predictive maintenance 
methods. (I.4.9.g) 


15 


5. 


A process exists to identify all applicable government safety and 
environmental regulations, including those concerning handling, 
recycling, eliminating, or disposing of hazardous materials. (1.4.9. b) 




o. 


Possession of appropriate governmental certificates indicating 
compliance to the identified applicable regulations. (I.4.9.b) 




7 
I . 


Work environment is clean and well-organized. (1.4. 9. b) 




8.1 


Evaluation areas for "appearance items" have appropriate lighting. 


20 


8.2 


Appropriate masters of "appearance items" are available. 




8.3 


"Appearance Item" masters and evaluation equipment are adequately 
maintained. 




8.4 


Verification exists that personnel making appearance evaluation are 
qualified to do so. 


25 


4.10 INSPECTION AND TESTING 




1.1 


Purchased material is controlled and verified per the selected system 
prior to release to production. (1.4.10.2) 




1.2 


Positive identification is provided for material used in production but not 
verified. (1.4.10.2) 


30 


1.3 


Where specified as the control method, suppliers submit statistical data. 
(L4.10.2) 
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2.1 


Product is inspected and tested as required by the documented 
procedures. (1.4.10.3) 




2.2 


Product is held until the required inspections and tests have been 
completed. (1.4.10.3) 


5 


2.3 


Defect prevention methods, such as statistical process control, error 
proofing, visual controls, is used rather than defect detection. (1.4.10.3) 




3.1 


Final inspection and testing is conducted in accordance with documented 
procedures. (1.4.10.4) 


10 


3.2 


Final inspection and testing ensures no product is shipped until all 
activities specified in the documented procedures have been 
satisfactorily completed. (1.4.10.4) 




4. 


Accredited laboratory facilities are used when required by the customer. 
(1.4.10.1,11.1) 




5.1 


Layout inspection is conducted per the Control Plan. (1.4.10.4) 


15 


5.2 


Functional testing is conducted per Control Plan. (1.4.10.4) 




6. 


Appropriate records are maintained for all inspections and tests. 
(1.4.10.5) 




4.11 INSPECTION, MEASURING, AND TEST EQUIPMENT 


20 


1. 


Inspection, measuring, and test equipment (including software when 
appropriate) has been provided that is capable of the required accuracy 
and precision. (1.4.11. 2. a) 




2. 


Required accuracy/precision of inspection, measuring, and test 
equipment is determined. (1.4.1 1.2.a) 


25 


3. 


Measurement system analysis is conducted (Gage R & R) for all gages, 
measuring, and test equipment, noted on the control plan. (1.4.1 1 .4) 




4. 


Appropriate criteria (per the Measurement Systems Analvsis Manuatt is 
used for acceptance of measuring equipment. (1.4.1 1 .4) 


30 


5. 


Where test software, hardware, or comparative references are used, 
capability and stability are verified prior to use (linearity and accuracy as 
appropriate). (I.4. 1 1 .2) 
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6. 


Each item of inspection, measurement, and test equipment is identified 
with a unique designation (including employee-owned equipment)? 
(1.4.11.3) 


5 


—7 

7. 


Each piece of inspection, measurement, and test equipment is calibrated 
at prescribed intervals and in the correct environment (including 
employee-owned equipment). (1.4.1 1.2.b) 




8. 


Gage condition and actual readings are recorded prior to recalibration. 
(1.4.1 1.2.e) 


10 


9. 


Appropriate actions, including customer notification, are taken on product 
and process when inspection, measurement, or test equipment is found 
to be out of calibration. (I.4. 1 1 .2.f) 




10. 


Inspection, measurement, and test equipment are properly handled, 
preserved, and stored to maintain calibration and fitness for use. 
(1.4.1 1.2.h) 


15 


A A 


Inspection, measurement, and test facilities (including software when 
applicable) are safeguarded to insure that calibration is not disturbed. 
(1.4.11.2.1) 




12. 


Records exist for recalibration of part-specific gages, etc. following 
engineering changes. (1.4.11.2) 


20 


4.12 INSPECTION AND TEST STATUS 




1. 


Inspection and/or test status is suitably identified throughout the 
production process. (1.4.12.1) 




2. 


If required by the customer, additional verification requirements are met 
for launching of new products. (1.4.12.2) 


25 


4.13 CONTROL OF NONCONFORMING PRODUCTS 




1. 


Systems ensure identification, documentation, segregation (where 
possible), to a designated area, and disposition of nonconforming and 
suspect product. (1.4.13.1) 


30 


2. 


Responsibilities for review and disposition of nonconforming and suspect 
product are clearly defined. (1.4.13.2) 
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3. 


Nonconforming and suspect products are reviewed according to defined 
procedures. (1.4.13.2) 


4.1 


Systems ensure nonconforming and suspect parts are reviewed to 
specified requirements. (1.4.13.2) 


4.2 


Systems ensure nonconforming and suspect parts are accepted with 
customer-approved concessions (EAPAs). (1.4.13.2) 


4.3 


Systems ensure nonconforming and suspect parts are reworked to 
approved repair standards. (1.4.13.2) 


4.4 


Systems ensure nonconforming and suspect parts are regraded for 
alternative applications. (1.4.13.2) 


4.5 


Systems ensure nonconforming and suspect parts are rejected or 
scrapped. (1.4.13.2) 


5. 


Processes assure that only material that has passed inspections and/or 
tests can be provided to the customer. (1.4.13.2) 


6. 


Nonconformances are recorded to permit defect analysis. (1.4.13.3) 


7. 


Reworked products are reinspected and/or tested according to the 
Control Plan. (1.4.13.2) 


8. 


Repair/rework instructions are accessible and utilized by the appropriate 
personnel. (1.4.13.3) 


9. 


Where applicable, approvals are obtained for products supplied for 
service applications that may have visible rework. (1.4.13.3) 


10. 


Systems ensure that customer authorization is received prior to shipping 
nonconforming material. (1.4.13.4) 


11. 


Records are maintained of the expiration dates for engineering approved 
product authorizations (EAPAs) and quantities authorized. (1.4.13.4) 


4.14 CORRECTIVE AND PREVENTIVE ACTION 


1. 


Appropriate corrective actions are developed to eliminate causes of 
nonconformances? (I.4.14.2.C) 


2. 


Disciplined problem solving methods are used. (1.4.14.1) 


3. 


Customer complaints and reports of nonconformances are effectively 
handled. (I.4.14.2.a) 
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4 


ociubea ui noricui uorTTiances are investigated and the results 
documented. (I.4.14.2.b) 




[zimoiivef icbb or corrective action is veriTieci. (I.4.14.2.G) 


6. 


Returned parts from customer's locations are analyzed and corrective 

aiAlUllo die IMIUcflloU. \H.£.) 


7. 


Nonconformance reports, (e.g. product quality, deviation, audit result, 
quality records, etc.) are used to develop preventive actions. (I.4.14.3.a) 


8. 


Relevant information on actions taken including changes to procedure 
are submitted for management review. (1.4.1 4.3.d) 


4.15 HANDLING, STORAGE, PACKAGING, PRESERVATION AND DELIVERY 


1 . 


Material handling methods prevent product damage and deterioration. 
(1.4.15.2) 


2. 


Storage areas are appropriate for preventing damage or deterioration of 
the product. (1.4.15.3) 


o 
O. 


When required by the nature of the product, the condition of product in 
stock is checked at intervals to detect deterioration. (1.4.15.3) 


4. 


Systems control the packing, packaging, and marking processes to the 
extent necessary to ensure product conformance to specifications. 
(1.4.15.4) 


5. 


Applicable customer packaging standards are available, (1.4.15.4) 


6. 


Compliance to applicable customer packaging standards. (1.4.15.4) 


7. 


Appropriate methods are used for product preservation and segregation. 
(1.4.15.5) 


8. 


Systems ensure the protection of product quality during delivery to the 
destination. (1.4.15.6) 


9. 


Appropriate analyses and corrective actions are used when scheduled 
delivery performance is not 100%. (1.4.15.6) 


10. 


Inventory management system optimizes inventory turns and stock 
rotation. (1.4.15.6) 


4.16 CONTROL OF QUALITY RECORDS 



30 



BNSDOCID: <WO 9952039 A 1J_> 



TABLE 1 


1. 


Records show effective operation of the quality system, including 

peraneni suD-conxractor quality recoras. (i.**.ioj 


2. 


Quality records are legible and readily retrievable. (1.4.16) 


3. 


Quality records (hardcopy or electronic) are stored in a suitable 
environment xo prevent deterioration, uamage, or loss. (1.4. lo) 


4.1 


Quality records are retained per established procedures. (1.4.16) 


4.2 


Quality records include production part approval, control charts, internal 
quality audits, and failure mode and effects analysis (FMEAs). (1.4.16) 


5. 


Quality records are available to the customer. (1.4.16) 


6. 


Systems ensure retention control and timely disposal of quality records. 
(1.4.16) 


4.17 INTERNAL QUALITY AUDITS 


1. 


Internal quality system audits are conducted as planned. (L4.17) 


2. 


Personnel conducting the audit are independent of the function being 
audited. (1.4.17) 


3. 


Audits are scheduled on the basis of the status and importance of the 
activity. (1.4.17) 


4. 


Audit results are documented and brought to the attention of the 
responsible personnel. (1.4.17) 


5. 


Corrective actions are timely, recorded, and evaluated for effectiveness. 
(1.4.17) 


6. 


Audits include work environment and general housekeeping. (1.4.17) 


4.18 TRAINING 


1. 


Training needs for all personnel performing activities affecting quality are 
met. (1.4.18) 


2. 


Qualifications for jobs affecting quality include identification of 
appropriate education, training needs, and experience. (1.4.18) 


3. 


Training records are maintained. (1.4.18) 


4. 


Training effectiveness is periodically evaluated. (1.4.18) 


4.19 SERVICING 
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1. 


Servicing meets the specified requirements. (1.4.19) 


2. 


Reporting and verification systems are established to communicate data 
from servicing functions to supplier manufacturing, engineering and 
design activities. (1.4.19) 


4.20 STATISTICAL TECHNIQUES 


1. 


Need of statistical techniques for establishing, controlling, and verifying 
the capability of process parameters and product characteristics has 
been identified. (1.4.20.1) 


2. 


Procedures are established and maintained to implement and control the 
application of statistical techniques. (I.4.20.2) 


3. 


Advanced quality planning is used to determine the appropriate statistical 
techniques. (I.4.20.2) 


4. 


Concepts of variation, control (stability), capability, and over-control are 
understood throughout the organization. (I.4.20.2) 


11.1 PRODUCTION PART APPROVAL PROCESS 


1. 


Complete supporting data exists for all production part approval 
submissions. 


2. 


Supporting data shows conformance to all customer requirements, 
including change notification. (1.4.9.6,11.1) 


3. 


Supporting data is organized and filed together for each part. 


4. 


Materials are purchased from customer approved subcontractor list. 
(1.4.6.1) 


II.2 CONTINUOUS IMPROVEMENT 


1. 


Continuous quality and productivity improvement efforts are a key 
element of the company's business. 


2. 


Specific improvement projects have been identified. 


3. 


Appropriate measurables are identified for improvement projects. 


4. 


Evidence of improvement in project measurables exists over relevant 
time periods (from six months to five years). 


II.3 MANUFACTURING CAPABILITIES 
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1. 


Cross-functional teams are used for facilities, equipment, and process 
planning in conjunction with the advanced quality planning process. 


2. 


Plant layout minimizes material travel and handling, facilitating 
synchronous material flow, and maximizing value added use of floor 
space. 


3. 


Mistake proofing techniques are utilized where appropriate. 


4. 


Resources are available for tool and gage design (If subcontracted, rate 
the tracking and follow-up system). 


5. 


Resources are available for tool and gage fabrication (If subcontracted, 
rate the tracking and follow-up system). 


6. 


Resources are available for complete tool and gage dimensional 
inspection (If subcontracted, rate the tracking and follow-up system). 


7. 


Resources are available for tool and gage maintenance and repair (If 
subcontracted, rate the tracking and follow-up system). 
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CLAIMS 

1 . A method of using a computer to gather information about an organizational process 

or system, comprising: 

displaying on a display device a plurality of questions, the questions being adapted 
to prompt an assessor to input on an input device of the computer the assessor's 
perceptions of the organizational process or system, wherein the assessor has at 
least some knowledge about the organizational process or system; 

receiving a plurality of inputs from the input device, wherein the inputs are stored in 
a memory; 

comparing within a processing unit of the computer the inputs to a plurality of values, 
and, if the inputs have first predetermined characteristics in relation to the values, 
then prompting the assessor to identify evidence supporting the inputs, and if the 
supporting evidence is identified, then validating the inputs for subsequent 
evaluation, and if the supporting evidence is not identified, then inhibiting 
validation of the inputs until the evidence is identified or until the inputs are 
changed to have second predetermined characteristics in relation to the values; 

analyzing the inputs to determine if one or more undesirable trends are present in the 
organizational process or system; 

displaying on the display device corrective action computer-driven questions, the 
corrective action questions being adapted to prompt the assessor to input on the 
input device the assessor's perception of the undesirable trend of the organizational 
process or system; and 

receiving a plurality of corrective action inputs, the corrective action inputs being 
stored in the computer's memory. 
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Fig. H 



Assessment Screen 







Rate: 

"How welldoes your; 
system address this 
issue?" 



Rating less 
than 50%? 



Yes 



Rating 
50% to 70%? 



Rating 70% 
or greater? 



Yes 



Yes 




Document name: 



Yes 



Next question for 
assessment 



Yes 




Save data and 
return to User 
Category Menu. 
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Fig. 6 



Assessment Screen 



Rate: 

"How would you rate 
the results achieved 
by the system?" 
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